
<!DOCTYPE html>
<!--[if IEMobile 7 ]><html class="no-js iem7"><![endif]-->
<!--[if lt IE 9]><html class="no-js lte-ie8"><![endif]-->
<!--[if (gt IE 8)|(gt IEMobile 7)|!(IEMobile)|!(IE)]><!--><html class="no-js" lang="en"><!--<![endif]-->

<head>


<meta charset="utf-8">
<meta http-equiv="cleartype" content="on">

<title>Install Flume on Local - Sam小龙</title>
<meta name="author" content="xiaolong.yuanxl">




<meta name="description" content="单机安装flume用于学习">

<meta name="keywords" content="flume ">


<!-- http://t.co/dKP3o1e -->
<meta name="HandheldFriendly" content="True">
<meta name="MobileOptimized" content="320">
<meta name="viewport" content="width=device-width, initial-scale=1">

<!-- Twitter Cards -->


<!-- Open Graph -->
<meta property="og:local" content="en_US">
<meta property="og:type" content="article">
<meta property="og:url" content="http://yuanxiaolong.github.io/blog/2014/08/17/install-flume-on-local">
<meta property="og:title" content="install flume on local">
<meta property="og:description" content="单机安装flume用于学习">

  <meta property="og:image" content="">

<meta property="og:site_name" content="Sam小龙">

<link rel="canonical" href="http://yuanxiaolong.github.io/blog/2014/08/17/install-flume-on-local">
<link href="/favicon.png" rel="icon">
<link href="/stylesheets/screen.css" media="screen, projection" rel="stylesheet" type="text/css">
<!-- <link href="//netdna.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css" rel="stylesheet"> -->
<link href="/stylesheets/font-awesome.min.css" rel="stylesheet">
<link href="/atom.xml" rel="alternate" title="Sam小龙" type="application/atom+xml">

<!-- the flow button on left -->
<script src="/javascripts/vendor/modernizr-2.6.2.custom.min.js"></script>


  <style type="text/css">body {background-image:url(/images/bg.jpg);background-size:cover;}</style>

<!-- 
  

  
    
  

  <style type="text/css">body {background-image:url(
      /images/
    
      bg.jpg
    
  
    );}</style>
 -->

<!--Fonts from Google"s Web font directory at http://google.com/webfonts
<link href="//fonts.googleapis.com/css?family=PT+Serif:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">
<link href="//fonts.googleapis.com/css?family=PT+Sans:regular,italic,bold,bolditalic" rel="stylesheet" type="text/css">
-->


</head>

<body id="post" >

<!--[if lt IE 9]><div class="upgrade"><strong><a href="http://whatbrowser.org/">Your browser is quite old!</strong> Why not upgrade to a different browser to better enjoy this site?</a></div><![endif]-->
<nav id="dl-menu" class="dl-menuwrapper" role="navigation">
	<button class="dl-trigger">Open Menu</button>
	<ul class="dl-menu">
		<li><a href="/">主页</a></li>

		<li>
			<a href="#">内容</a>
			<ul class="dl-submenu">
				<li><a href="/posts/">文章列表</a></li>
				<li><a href="/categories/">分类列表</a></li>
			</ul>
		</li>

		<li>
			<a href="#">关于</a>
			<ul class="dl-submenu">
				<li>
					<img src="/images/avatar.png" alt="xiaolong.yuanxl photo" class="author-photo">
					<h4>xiaolong.yuanxl</h4>
					<p></p>
				</li>
				<li><a href="/about">关于我</a></li>
				<li>
					<a href="mailto:232351936@qq.com"><i class="fa fa-envelope"></i> Email</a>
				</li>
				
				
				<li>
					<a href="http://github.com/yuanxiaolong"><i class="fa fa-github"></i> GitHub</a>
				</li>
			</ul><!-- /.dl-submenu -->
		</li>

		
	</ul><!-- /.dl-menu -->
</nav><!-- /.dl-menuwrapper -->




<div id="main" role="main" >
  <article class="hentry">
    <header class="header-title">
      <div class="header-title-wrap">
        
          <h1 class="entry-title"><a href="/blog/2014/08/17/install-flume-on-local/" rel="bookmark" title="install flume on local">install flume on local</a></h1>
        
        <h2>August 17, 2014</h2>
      </div><!-- /.header-title-wrap -->
    </header>
    <div class="entry-content">
      <p>介绍一下单机安装flume，并简单测试</p>

<!--more-->

<p>flume是一个数据采集的系统，可以实时获取产生的数据，并支持“推”和“拉”2种方式进行采集，且数据源也是多变的。</p>

<hr />

<h2 id="section">环境准备</h2>

<ul>
  <li>下载flume <a href="http://flume.apache.org/" target="_blank"><font color="#a97d43">官网</font></a></li>
  <li>如果要采集数据源到hdfs，需要安装hadoop</li>
</ul>

<hr />

<h2 id="section-1">配置</h2>

<p>1.生成配置</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line">xiaolongyuan@xiaolongdeMacBook-Air flume-1.5.0<span class="nv">$ </span><span class="nb">cd </span>conf/
</span><span class="line">
</span><span class="line">xiaolongyuan@xiaolongdeMacBook-Air conf<span class="nv">$ </span>ls
</span><span class="line">flume-conf.properties.template flume-env.sh.template          log4j.properties
</span><span class="line">
</span><span class="line">xiaolongyuan@xiaolongdeMacBook-Air conf<span class="nv">$ </span>cp flume-conf.properties.template flume-conf.properties
</span><span class="line">xiaolongyuan@xiaolongdeMacBook-Air conf<span class="nv">$ </span>cp flume-env.sh.template flume-env.sh
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>2.修改 <font color="green">flume-env.sh</font> 、<font color="green">flume-conf.properties</font></p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span>flume-env.sh</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line"><span class="nv">JAVA_HOME</span><span class="o">=</span><span class="sb">`</span>/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home<span class="sb">`</span>
</span></code></pre></td></tr></table></div></figure></notextile></div>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span>flume-conf.properties</span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
<span class="line-number">10</span>
<span class="line-number">11</span>
<span class="line-number">12</span>
<span class="line-number">13</span>
<span class="line-number">14</span>
<span class="line-number">15</span>
<span class="line-number">16</span>
<span class="line-number">17</span>
<span class="line-number">18</span>
<span class="line-number">19</span>
<span class="line-number">20</span>
<span class="line-number">21</span>
<span class="line-number">22</span>
<span class="line-number">23</span>
<span class="line-number">24</span>
<span class="line-number">25</span>
<span class="line-number">26</span>
<span class="line-number">27</span>
<span class="line-number">28</span>
<span class="line-number">29</span>
<span class="line-number">30</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line"><span class="c"># The configuration file needs to define the sources,</span>
</span><span class="line"><span class="c"># the channels and the sinks.</span>
</span><span class="line"><span class="c"># Sources, channels and sinks are defined per agent,</span>
</span><span class="line"><span class="c"># in this case called &#39;agent&#39;</span>
</span><span class="line">
</span><span class="line">localagent.sources <span class="o">=</span> localSource
</span><span class="line">localagent.channels <span class="o">=</span> localChannel
</span><span class="line">localagent.sinks <span class="o">=</span> localSink
</span><span class="line">
</span><span class="line"><span class="c"># For each one of the sources, the type is defined</span>
</span><span class="line">localagent.sources.localSource.type <span class="o">=</span> netcat
</span><span class="line">localagent.sources.localSource.bind <span class="o">=</span> localhost
</span><span class="line">localagent.sources.localSource.port <span class="o">=</span> 44444
</span><span class="line">
</span><span class="line"><span class="c"># The channel can be defined as follows.</span>
</span><span class="line">localagent.sources.localSource.channels <span class="o">=</span> localChannel
</span><span class="line">
</span><span class="line"><span class="c"># Each sink&#39;s type must be defined</span>
</span><span class="line">localagent.sinks.localSink.type <span class="o">=</span> logger
</span><span class="line">
</span><span class="line"><span class="c">#Specify the channel the sink should use</span>
</span><span class="line">localagent.sinks.localSink.channel <span class="o">=</span> localChannel
</span><span class="line">
</span><span class="line"><span class="c"># Each channel&#39;s type is defined.</span>
</span><span class="line">localagent.channels.localChannel.type <span class="o">=</span> memory
</span><span class="line">
</span><span class="line"><span class="c"># Other config values specific to each type of channel(sink or source)</span>
</span><span class="line"><span class="c"># can be defined as well</span>
</span><span class="line"><span class="c"># In this case, it specifies the capacity of the memory channel</span>
</span><span class="line">localagent.channels.localChannel.capacity <span class="o">=</span> 100
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>其中如何配置，更多信息需要参考<a href="http://flume.apache.org/FlumeUserGuide.html" target="_blank"><font color="#a545aa">user guide</font></a></p>

<hr />

<h2 id="section-2">运行</h2>

<p>1.在flume的bin文件夹，运行下面命令，可以看到已经成功在44444端口启动服务</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
<span class="line-number">2</span>
<span class="line-number">3</span>
<span class="line-number">4</span>
<span class="line-number">5</span>
<span class="line-number">6</span>
<span class="line-number">7</span>
<span class="line-number">8</span>
<span class="line-number">9</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line">xiaolongyuan@xiaolongdeMacBook-Air bin<span class="nv">$ </span>flume-ng agent --conf ../conf --conf-file ../conf/flume-conf.properties --name localagent -Dflume.root.logger<span class="o">=</span>INFO,console
</span><span class="line">Info: Sourcing environment configuration script /Users/xiaolongyuan/Documents/flume-1.5.0/conf/flume-env.sh
</span><span class="line">Info: Including Hadoop libraries found via <span class="o">(</span>/Users/xiaolongyuan/Documents/hadoop-1.2.1/bin/hadoop<span class="o">)</span> <span class="k">for </span>HDFS access
</span><span class="line">Warning: <span class="nv">$HADOOP_HOME</span> is deprecated.
</span><span class="line">
</span><span class="line">……省略
</span><span class="line">
</span><span class="line">2014-08-17 21:08:08,900 <span class="o">(</span>lifecycleSupervisor-1-0<span class="o">)</span> <span class="o">[</span>INFO - org.apache.flume.source.NetcatSource.start<span class="o">(</span>NetcatSource.java:150<span class="o">)]</span> Source starting
</span><span class="line">2014-08-17 21:08:08,914 <span class="o">(</span>lifecycleSupervisor-1-0<span class="o">)</span> <span class="o">[</span>INFO - org.apache.flume.source.NetcatSource.start<span class="o">(</span>NetcatSource.java:164<span class="o">)]</span> Created serverSocket:sun.nio.ch.ServerSocketChannelImpl<span class="o">[</span>/127.0.0.1:44444<span class="o">]</span>
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>2.利用telnet命令，输入任意字符</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line">telnet 127.0.0.1 44444
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>3.可以看到flume已经“感知”到了有数据发送过来了。</p>

<div class="bogus-wrapper"><notextile><figure class="code"><figcaption><span></span></figcaption><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class="line-number">1</span>
</pre></td><td class="code"><pre><code class="bash"><span class="line">2014-08-17 21:09:26,936 <span class="o">(</span>SinkRunner-PollingRunner-DefaultSinkProcessor<span class="o">)</span> <span class="o">[</span>INFO - org.apache.flume.sink.LoggerSink.process<span class="o">(</span>LoggerSink.java:70<span class="o">)]</span> Event: <span class="o">{</span> headers:<span class="o">{}</span> body: 74 68 69 73 20 69 73 20 6D 79 20 66 69 72 73 74 this is my first <span class="o">}</span>
</span></code></pre></td></tr></table></div></figure></notextile></div>

<p>flume还可以将数据源的信息，直接写入到hdfs等等，需要进一步查看官网文档</p>


      <!-- add copyright for each post -->
      <div style="margin-top:20px;">
        <font color="gray" size="3">
          原文地址，转载请注明出处:<br/>
          <a href="http://blog.yuanxiaolong.cn/blog/2014/08/17/install-flume-on-local/">http://blog.yuanxiaolong.cn/blog/2014/08/17/install-flume-on-local/</a>
        </font>
      </div>
      <footer class="entry-meta">
        <span class="entry-tags"><a href="/categories/#flume" title="Pages tagged flume" class="tag">flume</a></span>
        <span><a href="/blog/2014/08/17/install-flume-on-local/" rel="bookmark" title="install flume on local">install flume on local</a> was published on <span class="entry-date date published updated"><time datetime="2014-08-17T20:46:58+08:00">August 17, 2014</time></span></span>
        
        <span class="author vcard"><span class="fn"><a href="" title="About xiaolong.yuanxl">xiaolong.yuanxl</a></span></span>
        <div class="social-share">
          <div class="socialcount socialcount-small inline-list">
            <!-- <li class="facebook"><a href="https://www.facebook.com/sharer/sharer.php?u=/blog/2014/08/17/install-flume-on-local/" title="Share on Facebook"><span class="count"><i class="fa fa-facebook-square"></i> Like</span></a></li>
            <li class="twitter"><a href="https://twitter.com/intent/tweet?text=/blog/2014/08/17/install-flume-on-local/" title="Share on Twitter"><span class="count"><i class="fa fa-twitter-square"></i> Tweet</span></a></li>
            <li class="googleplus"><a href="https://plus.google.com/share?url=/blog/2014/08/17/install-flume-on-local/" title="Share on Google Plus"><span class="count"><i class="fa fa-google-plus-square"></i> +1</span></a></li> -->
            <!-- 多说评论框 start -->
          	<div class="ds-thread" data-thread-key="/blog/2014/08/17/install-flume-on-local/" data-title="install flume on local" data-url="http://blog.yuanxiaolong.cn/blog/2014/08/17/install-flume-on-local/"></div>
            <!-- 多说评论框 end -->
            <!-- 多说公共JS代码 start (一个网页只需插入一次) -->
            <script type="text/javascript">
            var duoshuoQuery = {short_name:"yuanxiaolong"};
            	(function() {
            		var ds = document.createElement('script');
            		ds.type = 'text/javascript';ds.async = true;
            		ds.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + '//static.duoshuo.com/embed.js';
            		ds.charset = 'UTF-8';
            		(document.getElementsByTagName('head')[0]
            		 || document.getElementsByTagName('body')[0]).appendChild(ds);
            	})();
            	</script>
            <!-- 多说公共JS代码 end -->
        </div>
        </div><!-- /.social-share -->
      </footer>

    </div><!-- /.entry-content -->
    
      <div class="read-more">
        
          <div class="read-more-header">
            <a href="/blog/2014/09/07/install-nfs-and-share-file/" class="btn">Read More</a>
          </div><!-- /.read-more-header -->
          <div class="read-more-content">
            <h3><a href="/blog/2014/09/07/install-nfs-and-share-file/" title="install nfs and shshare file">install nfs and shshare file</a></h3>
            <p>安装nfs服务器，用于共享配置 <a href="/blog/2014/09/07/install-nfs-and-share-file/"> Continue reading</a></p>
          </div><!-- /.read-more-content -->
        
        <div class="read-more-list">
          
            <div class="list-item">
              <h4><a href="/blog/2014/09/07/install-dns-server-bind/" title="install dns server bind">install dns server bind</a></h4>
              <span>Published on September 07, 2014</span>
            </div><!-- /.list-item -->
          
            <div class="list-item">
              <h4><a href="/blog/2014/08/31/analysis-hadoop-load-configuration/" title="analysis hadoop load configuration">analysis hadoop load configuration</a></h4>
              <span>Published on August 31, 2014</span>
            </div><!-- /.list-item -->
          
        </div><!-- /.read-more-list -->
      </div><!-- /.read-more -->
    
    
  </article>
</div><!-- /#main -->

<div class="footer-wrapper">
  <footer role="contentinfo">
    <span>&copy; 2014 xiaolong.yuanxl. Powered by <a href="http://octopress.org">Octopress</a> using the <a href="https://github.com/Z1MM32M4N/hpstr-theme/">HPSTR Theme for Octopress</a>.</span>

  </footer>
</div><!-- /.footer-wrapper -->



<script src="http://cdn.staticfile.org/jquery/1.9.1/jquery.min.js"></script>
<script>window.jQuery || document.write('<script src="/javascripts/vendor/jquery-1.9.1.min.js"><\/script>')</script>
<!--
<script src="/javascripts/octopress.js" type="text/javascript"></script>
-->
<script src="/javascripts/scripts.min.js"></script>
<!-- 
 -->
<script type="text/javascript" src="http://tajs.qq.com/stats?sId=35560703" charset="UTF-8"></script>






</body>
</html>
